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In somo araa^ of psychology (particularly Social and 
Cognitive Psychology) introductory psychology students have 
been the primary seurae of data collection* In 1977, 73 .S 
puresnfc of tho articles In th^ Journal of P ersona 1 ity and 
S oc t a 1 P &y c h o logy and 92*7 percent of th© articles in tha 
Journal of E>cp€j.r.i mental Psychology: Perception udad 
introduotory psychology students (Kulich, Geidon, Richmrdaan 
& S&rvl£!i s not© S)* Sueh statistics seem surprising in 
light of the criticisms regarding the validity and 
gonoral i ^abi 1 i by of data from sueh a population (Orne, 1962, 
1969; Rosenthal & Rosnow, 1989), Others have questioned the 
potentially oo^rciv^ nature of introductory psychology 
sub j eot poo Is (Cool*, Kimble, Hicks, MeQuiro, Sohoggcm , & 
Smith, 1971)- The questions of ethi cs and sc ient i fie value 
demand that subject pool usages be carefully serutini sad. 
Recent prob 1 ems with introductory psychology subject pools 
in Nav York ( i , e . , being shut down due bo failure bo 
adequate 1 y scrutinize a research study) suggests that such 
scrutiny may come ex terna 1 1 y if not internally. 

Evaluation research examines the effects of policies 
and programs on their targets in terms of the goals they are 
meant to achieve, The major assumption of ova ! \;&t ion 
research is that providing empirically derived information 
relevant to decision making results in better decisions theui 
decisions resulting solely from speculation, Such an 
evaluation research project generally begins with the 
identification of goals re I abed to tha policy or program. 
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Tha .goft Is of the introductory pnyohol ogy .pq.gt 

Jung (1859) surveyed SO universities! aiakihtf l*or th«* 
purpa^a and jusbi fixation for the inbroduefcery psychology 
pools* Siafcy^four pereenb of bho uni vers ifcio^ aurvsycd 
responded that ib§ purpose was both educational and ba ©orvc 
department t\m%dn* Thi rty-bhr©o peroonb responded that its 
justification wao educational only* while only 2 p©ro«§nt 
responded department ne^dD only. 

Since it Is unlikely that many researchers run 
©Kperimanfco solely for the b€nefit of the subjects; it is 
evident that a central goal from the researchers viewpoint 
and displayed as "departmental needs* in the rssponoas from 
tha universities i^ faho advancement of soienee through the 
co 1 lection of data from rsadily obtainable subject*}* 

Nineby^seven percent of the universities rtvi^wed 
responded with education as a justi f ieab ion for introductory 
psychology subject pools. It is the arifcical diff^rtnoe 
between coercive m%p 1 oibat ion of students and combining 
education with scientific inquiry. Thus it i s an apparent 
second goal * 

The American Psycho 1 ogiesU Association < 1973 ) has 
pub 1 i shed a set of ethical principles for research. Most 
programs engage in peer review to insure ethical research . 
'T'ie first two ^oa 1 s must be accomplished while in compi iance 
with ethical standards. 

While other goal s could certainly be suggested, such as 
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«du«atian of graduate studonta, generating publications, and 
protection of th« existence of th<# Duhj#ot pool, which may 
or* may not bo subsumed under compliance with ethical 
standards, it is suggested that those are the primary coal^ 
of th<a introductory psychology nubjeob pool pfograifts* with 
ofch^r jjsale subsumed within these goals* 

Evaluation of th<a attainment of these goalo entails tho 
as3€£srnt3nfe of information already reported and the 
development and application of measures to further assess 
th© value of such programs in r%prds bo each goa 1 . 

Advancement ef science 

One need only look at a few issues of the Journal of 
Personal ifay and Social P syehalosy to see that introductory 
psychology pools play a large role (72. 3 percent) in the 
pub lication of articles (Kulich, eb al . , 1978)* However, 
whether the importance of the subjeet pool is proportional 
to its representation in journal articles is subject to 
serious question* 

Evidence indicates that sub j ecbs from introductory 
psychology pools differ from the general population on 
numerous dimensions <Be 1 1 , 19S2 * Smart, 1966). It has even 
been suggested that introductory psychology students are not 
eyan representative of college students of the same year in 
ool lege (Rosenthal & Rosnow* 1969), Even within the subject 
pool itself, biases related to the subject participating in 
tha study early or late in the semester (Seldon, Kulich & 
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Hsaty, 1978), experiene© .in experiments (Silverman, 
Shulman, SWieaonthal, Hoimos, 1987) and slightly 

increased levels of sophistication of the subjects (Pagd , 
1988, 1909) have been found, lb is thus not surprising that 
Rosenbhal and Reshow (1989) have suggested that MeNemars 
proposal <1948) that, "bho existing seieneo or human 
behavior is largely bho science of the behavior of 
sophomores" sounds unduly optimistic, 

WhUe the external validity of exper indentation with 
Introductory psyebolo|y subject pools is open bo question, 
the §am« is true of many alternate samples. Volunteers have 
been demonstrated to bo different on dimensions such as 
birth order (Capra and Dittos, 1962), level of approval 
seeking (MeDavld, 196B) and various other dimensions (Bell, 
1982; Lasogna and Von Felsinger, 19B4; Riggs and Kaess, 
1955), When paid volunteers are used, subjects are found to 
have a higher need for cash (Howe, 19Q0) , which can be 
expected to be confounded with social class. 

Without coercion as a method of recruitment, external 
validity may be a question in all future experimentation. 
As a control variable, the method of recruitment becomes a 
problem when it is not controlled at various levels, i.e. 
the failure to replicate research with other recruitment 
methods. The extent to which subject pools provide sstsily 
accessable samples thus diminishing such replication may 
limit its value to the advancement of science. 
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Tho Internal validity of exper imt&nkatsien with 
inbreduefcery psychology students has also b<s«n quofsbien^d » 
Pago has found bh& lovel of dophi^tioat ion of subjeobs bo 
confound c lassieal conditioning of attitudes (Page, 19©fi) 
and f igurd-greund perception experiments (P^gg t 1908)* The 
increased sophistication of psychology students bhreugh 
coursD parbicipabion may inoraa&a effects duo bo demand 
eharaober lab ios . Weber and Cook C 1072) have out I ined four 
dornand characteristic related roles for subjects , The good 
and the negabivisbia subjects bry bo confirm or disconfirm 
the experimenters hypobhesis, The increased sophistication 
of subjects may be related bo guessing the hypothesis and 
attitudes toward required service in experiments may 
determine the motivation fco confirm or disconfirm the 
hypothesis* A third role suggested by Weber and Cook, the 
evaluative subject, may appear more frequently if the 
sub jecb relates participation bo course evaluation. 

Other means of subject recruitment may make other 
threats to internal validity more salient* Vo lunteer s , 
which have been shown ta be more approval seeking (McDavi d , 
1965), may be more suseeptable to being the evaluative 
subject. Unfortunately, the easy accessab: 1 iby of 
inbroduetory psycho 1 ogy poo I s reduces the frequency of 
replications in samples whose suseeptibi 1 ity to demand 
characteristics differ. 

It could be suggested that the accessabi 1 ity of 
subjects may be both th^i chief strength and weakness of 
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inbroduebory psychology pools. It undoubtedly Increases the 
quantity of 1 1 l*-eensidared r©iieeii*oh baeituoa ib requires 
litbla rocruitmahb effort* 

A bobal asaaesmanfa of the Impact of the inbroduebory 
psycho 1 ogy poo 1 on psycho 1 ogy ^oems somewhab 1 ois bhan 
feasible. However, bha ongoing a^ossmant of factors within 
bho pool aosaoiabad with bha adsmnetimanb of soiance can be 
oxpeobed bo improve decision making whare ib affects bho 
pool's value bo bha science. 

Enper imenfaal requirements ai a pedagogic endeavor 

Of fifby-two univarsibias surveyed (Jungi 1989)* 97 
percent responded thab educational value i^as a justification 
for requiring the participation of psycho logy students in 
experiments, Yet 38 percent of these uni versities required 
no feedback of the methods and purpose of the experiment , 
Many mora did nob require bha feedback bo b© immediate, thus 
requiring bha subject to return if he desires such feedback, 
While ib is o 1 ear that universities wish bo portray 
experimental participation as an educational endeavor > ib is 
probably nob a key consideration in maintaining chat pool. 
Principle mi% of bha American Psycho 1 ogica i 
Associations 's ethical guidelines (Ad hoc Committee on 
Ethical Standards in Psychol ogioal Research, 1973) pn-r tains 
to bha eduoabional value of experiments, 

"Ethically acceptable research begins with 
the establishment of a c 1 ear and fair 
agreement between the in vesfci gattor and th«i 
research participant that clarifies tha 
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responsibilities of each, Tho investigator 
has the obligation to honor all premises and 
sommitbments included in that agreemonb," 

Inclusion of the research participation requirement in 
course dasor iptionn » its requirement as a course component, 
and most likely the manner in which it is presented 
explicitly or implioibly suggests that the subject's 
assistance will be repaid with a contribution to the 
student ' s psychological know! edge . Failure to do this is 
clearly ineensisbenb with the prino ip 1 e of a fair agreement 
as stated in principle eiK and unless some other form of 
equitable payment is giveni probably represents an abuse of 
power as educators and controllers of grades for personal 
benefit or the benefit of one 's colleagues, 

If m professor gave two eKtra credit points for every 
three dollars a student gives him or gives an incomplete to 
any students who fail to give him a set amount of money and 
then used that money to hire subjects , there is little doubt 
that most would consider this act unethical* It is unlikely 
that many programs allow students to hire others to do their 
experiments for them. The student is required to 
participate in the experiments because when credit for the 
course is awarded* a set of knowledge is expected to have 
been collected by the student* Yet many programs make 
feedback optional for the experimenter f subject or both * 
Testing generally insures that learning is taking place and 
provides motivation to the student f yet measurement of this 
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i earning in rarely eneouhfeertidi Researchers hav«i be#n heard 
suggesting feho students would rather leave than hear bh^ 
feedback, Yab wh«r© else do we giv<a caursa orsdib without 
requiring bhe aubjeeb bo abbend bo fahn e*ohbant of a 
component of a eeurse? 

A poggibla assumption regarding research participation 
is that mora exposure rosulba in ^oma form of Uarning, 
What kind of learning, how muoh learning and wh^fchmr that 
learning is relevant be the purposes of the introductory 
psychology coursa has never been evaluated* Thi s mere 
exposure hypobhesis has oerbainly not bean accepted by many 
academi cians sines testing continues, to be prevalent in most 
courses, Evaluation of mere exposure learning and possible 
alternatives is critical bo the educational justification of 
introductory psychology pools* 

Th^ American Psycho 1 ogioa 1 Association 'B Ad Hoc 
Committee Ithiaal Standards in Psychological Research (1973) 
recommends continuing evaluation of student attitudes toward 
experiment participation in a set of recommendations for 
conducting subject pools. If education as a just i f ication 
for pools is to continue, evaluation of experimental 
participation for educational value and camparisions of 
methods of administration are needed. Yet journal 
discussion of evaluation reveals only one limited evaluation 
of a new system (Davis and Fernald, 1975), Davis and 
Fernald ' note that the number of required experiments is 
frequently tied to research needs rather than the needs of 
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the ebudenba. Sinee suoh a pfa«fcies suggestr inoonsistehey 
with ebhleal principles and publto jusbif icafciona , bhey 
reeoinmend demonstrations bo augmont experiments and the 
maihbainina of a oendbanb research requi remenb . 

The dearth of informabion as bo the af reebiveness of 
methods of conducting subjeeb ji^li makes it impossible td 
draw eoneUaions about the value of axp«r Imenbal 
parbieipabien for the subject, Early investigations 
(Miller, Kul loh and Hanson, note Si S*Ucn. 
Kulich, Richardson and Sorvies, nob« 3) suggest that 
subjective evaluations of students regarding the worth of 
bhs oxperimenbal part ie Ipation is varied bub slightly 
negative. The objectives of axporlmenbal participation 
require c lart f leation before adaquabo evaluation measures 
can be developed. Consistent method* and measures could 
allow comparisiena and suggest systems of greater value to 
the student, 

Cowpl ianee wibh ethical sfaandarris 

Jung (ISSS) found bhat over 90 percent of all subjects 
used were college sbudents, 48,0 percent participated 
because it was required, S5.4 percent as an option and 22.1 
percent for extra credit. The initial draft of echieal 
principles (Cook. Kimble, Hicks, MeGuire, Sohoggen, and 
Smith, 1971) states: 

"Students should not be required to participate in 
research as a condition for entering a course or 
for obtaining grade points or avoiding loss of 
thaw, or as an alternative to another onerous-, 
baslt, where that participation requirement is to 
any extent in the service of research. 
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Although thia prinaipU was fiat adodjpbed In bhi Flml 
draft, subjiat j>ools eohtfeobed with the ©nre U in#mt ih 
aourti^ tktn givah gp^oial aantldiPabiOil it\ bhi After iti&ft 
Psyehe logical Asseaiat ioh 1 s ftthi^al gUidtl£n«# (Afterisiaft 
Psyeholog Association I Ad hoa Oeffifiiitbei on Ebhia&i 

Sh^ndardfe in Pftyeh&UtfitM R«i3*ii\rsh* li?75> dun to it& 
potential ly ea«retv# imfeur*! and pabenttally qu^st ienablfe 
Justification ih bmjrmm 6f value bo fch^ subj^dt&t Eight 
reeomnieftdabioftd hfm £>ropog£d» 

Ths firitb r eoomrrt^tfidiib ien £Ugg%§bg that the «Kp^rim%ntiil 
participation requirement b£ ineludod in official oourso 
listings* tfnf orbunabe ly , the value of gueh insluaion i£ 
lo£b wh#n tha eoupn© U r^cjuired* Further* if nab required, 
it denial educational opportunity Tor thos§ who ara 
inb#r«sbbd tn thci eeur^e* Thus suoh a practise is of llfafaia 
val us* 

The second recommendation is for r#viav by oomtsi ttefis 
bo protect subjects. The ooramitfc©% g^n#rally %n^ag€a in 
ariftahair speculation f€gardinfi potential risk, a practico 
which th<§ rigorous scientist generally frowns upon* Garg%n 
(1973) has suggested that Tactual advic£ ,, is much mor<i 
beneficial than codified ethice. Subjects have ho«n asked 
to subjeetively rate different basks for avarsiveness (Farr 
and Seaver, 1975) , ethics <Sullivan and DmtUmr > 1973* Wilson 
and Donnerstein , 1978), and legality (Wilson and 
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Donnarstein » 1976) and l%gsl advisers have given cpinLon^ *«ft 
tile legality of some manipuiafci onm <SiI versiart* 1^75)* TSia 
ability of debriefing te diffuse nejafcive fee llx\g^ i^imaisns 
unclear and complex (Ring, Sfalston and Corey^ 11370; W^lsfcs^, 
Berscheid, Abraham and Aronsen , 1SST}* Methods %asi njf r^l<e~ 
pitying fco anticipate inf orirted consent in de^fpti^n studies 
^rhtoh could potentially ba ivirsive hmy^ b-iei^ ^ugvfe^t^d 
(Berscheid, Baron, De>rEier and Libyan, 1973) m R^l^p laying 
sterns promising as a first step In the asses^fa^Tit of* risk dn 
a research design- However, fch^ predictive a.bi Uisy or such 
vicarious evaluation has yet t«3 be well substantiated. , T&a 
speculation of subjects on tlteix reaction to a m^ni pu_ 1 at ion 
in ah experimental situation may be mare pr*id^ofeiv*e ^f ttie 
rev iev committee ' s speculations than actua 1 reactions. 

Post-experimental aiiesittiint as done by Abracosoii (l&TP) 
in regards to research on sema li ty necessitates ^xp-oSTire of 
the subjects prior to assessment. Daveloplni and testing 
models for the prediction or sabject * s reac^ti-ons and 
evaluation of reactions after the ixperitotnt ^uN al-low 
assessment of various procedures and the eff ^afcivr^n^t^ of 
subject usage committees* 

Reeomniendations three and four ©f the Ad hoc 0^mmi&tm% 
on Ethical Standards in Psychol ogiccai Research deal s v^ifah 
informed consent, as did itea ones. Item three reconim^TiSfi 
alternate projects' for fchos^ who prefer not to part deip^t®, 
Alternate projects can unfortunately hu madt quiter <^n«r^u^ f 
thus negating their value, This author has hsU ^nly erit§ 
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student Mghfe hundred ohoose an alternate project <an 
alternata project urhioh was imposed and not by the authors 
choice)* ^suiMesting that it may not be a viable alternative, 

I tela tlhrea also suggests that subjects choose the type 
of %^pmrimmnt in whieh they partioipata, Subnet's time 
1 imitations* lack of insurance that numerous experiments art 
available and minimal descriptions of the experiment ean 
strv% to snake such a choice impossible, 

ttm% four suggests that subjects be reminied of their 
right to JeaLVe at any time and mentions the signing of an 
informed 0onsenfe after an exp lanation of the procedures* A 
reeeritly performed experiment used such procedures* 
Subjects were informed upon arrival that their would sing 
before at jaft^l of music experts and wmrm asked to sign an 
informed consent * The same individuals f or whom: singing in 
front of an audience is traumatic (high so-ciaj anxiety) 
would be the individuals who might be fearful of refusing to 
participate. Handled well by the eHper imetiter , this dilemma 
could be abated to a large degree. Unfortunately, whether 
the obtainment of informed consent and the post-experiment 
debriefing are effective is difficult bo evaluate prior to 
the actual running of subjects. Continuous post* 
aaperimen tat! feedback might b# the best assessment 
procedure* A olearer image of whether and how demand 
character istios prevent subjeets from droppinff out of the 
experiment demands further investigmt ion. 
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Itmm fiv© deals with treating subjects with respect and 
courtesy, £, e . , remembering that this white rat is human . 
It%m si x discusses rewards for risoarch such iai education, 
This item is quite central be the goals of the subject pool 
and has already been dealt with in greater detail* 

Items sevdri and eight deal with evaluation. The 
seventh recoimneiidafcion suggests a mechanise Tor student 
complaints* 0n# oould suggest that such a feedback system 
is much too limited and places the onus of evaluation on the 
subject whose only motivations may be al truism bo protect 
others or, morm likely, revenge* This inducement may by 
insufficient for many dissatisfied subjects* 

Item eight calls for continuous evaluation and 
improvement of recruiting procedures, It might be advisable 
that such avmluafcioft be eKtended to include educational 
value and the effectiveness of screening procedures , 
However the criticism of the lack of evaluation is certainly 
easier than the development and analysis of measures 
necessary for such evaluation, 

V/hy a dec isi on-theoretic approach? 
Given that the goals of the introductory psychology 
subject pools can be identified, and the attainment of these 
goals can be assessed, how can the information be organised 
such that it will improve decision making? The two 
alternatives ^fhioh are available are multivariate hypotheses 
testing and the dec isi en^theoreti o approach as presented by 
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Edwards mt ai*(197B>* 

Hypothesis besting bulls Mm whether it is 1 ikely that a 
di f-f erenc* occurs by change. Essentially, since tvo 
al ternati ves cannot be identical • it talis us it is likely 
that we have successful 1 y measured a difference in bvo 
alternatives. It does little to aid in the organisation of 
information and tails us little about the level of impact of 
an alternative. Its philosophy suggests that we learn 
nothing from a probability of .15 and allows the luxury of 
confidence if the probability falls below - 05, The 
experimentalists approach x% bast suited to an unchanging 
program in which subjects , committees , reviewers, 
experimenters and other factors remain constant* It is 
unlikely that any department can adjust its subject pool 
policies to provide a suitable atmosphere for traditional 
experimental methodo I of y * 

Edwards, fluttentmf, and Snapper (1975) discuss the 
"baseball stati sti c ians approach"* The experimentalist 
wishes to tesfe a hypothesis giving one a simple dichotomous 
result*, The baseball statistician wants to describe a 
phenomena as thoroughly as possible and present that 
information in a fashion that is usable for the manager. 
Further* such evaluation is formative rather than summative 
in nature, i.e. , it allows continuous evaluation and 
feedback for the utilisation of decision makers, rather than 
'the evaluation of an end product* It is able to cop* with 
the continuous change which generally occurs in most 
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programs, Thus* such an approach seems best suited to our 
needs * 

Applying the decision— theoretic approaeh 
The major goals of the introductory psychology subject 
pools have already been disuessed. Each of these three 
goals can best be linked to different but nonexclusive 
subgroups. It is expected that researchers are most 
concerned with the advancement of science , introductory 
psycho logy instructors are most concerned with educational 
value arid the subject usage committees are concerned with 
ethical eons i derations • 

From each subgroup , subgoals which exist within each 
goal can be assessed* Under the advancement of science may 
be subgoals such as training of graduate students, 
generation of publications, obtaining pilot data and 
obtaining significant findings. Under educational value one 
might find the ability to identify independent and dependent 
variables* understanding of .experimental rationales or a 
clearer and more real ist ic perception of the meaning " "of 
laboratory exper imentation, Subgoals regarding the ethical 
principles may include the feeling of freedom to decline 
participation* minimal stress associated with 

experimentation and the lacking of undesirable after effects 
of experimental participation. Measures for each of the 
subgoals must then b# devised. 

Step two involves the assessment of the relative 
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isiportanee of each- goal. Since the evaluation of all goals 
relative to ea,oh other would be extremely eumbifiemi , the 
relative importance of subgoals within eaoh primary goal 
will bt assessed followed by the assessment of relative 
importance of each sutjioal « The group most strongly linked 
to each goal (researchers » educators and subject usags 
committees) vi 1 1 assess relative importance as outlined by 
Edwards at, a.1 , (1978), The least important goal is 
assigned a value of ten* with other goals evaluated relative 
to the least important goal. The same process-,™ sampling 
from a combination of the above subgroups, gives the 
relative importance of each major goal. 

Step three requires the establishing of criteria for 
foal attainment, Measures vh i c h are dichofconieus are most 
suitab le since they require no oriteria judgement. Either 
the response is desirable or undesirable* Non-dichotomous 
measures require a conversion to a dichotomy for utilisation 
as probabilities of goal attainment. 

Prior probabilities for goal attainment must then be 
estimated* These prior probabilities are generally based on 
the subjective probabilities of those most closely related 
to the goal being assesed, To the extent that these 
subjective probabilities are formulated from previously 
obtained data, these prior probabilities could reflect data. 
For many or most , they are speculative. Where different 
subject pool policies or methods are being compared, these 
prior probabilies would reflect - expected differences and 
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provide a means of viewing al i considerations when proposing 
alternatives . 

Prior utilities can be oaleu Iated based on prior 
probabi 1 ities , yiel ding a projected utility for 
alternatives * The ut i lity of each goal can be found from ; 

n 

i=l (1) 

where Uj — the utility of each goal* P £ represents the prior 
probability of subgoal attainment, = the weighting for 
that subgoal s and n » the number of subgoals within the 
goal* The prior utility can then be found as follows: 

3 

U ^ 2 UjWj 

j-i <2) 

where U the utility of the system and Wj - the weighting 
of the three goals. 

It is this point at which one collects data, and 
converts it to percentages of goal attainment* From these 
percentages for attainment of goals, posterior probabilities 
of goal attainment can be obtained using an algorithm 
outlined by Edwards* Landmen t and Savage (1963) using the 
beta distribution as a prior and, since at this point in the 
evaluation of subject pools these prior probabilities are 
weakly held, assuming that. the parameters of the prior beta 
distribution sujti to 1.0, An example of the utilisation of 
this procedure can be found in Edwards, Guttentag and 
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Snapper (1975). 

From the posterior prebabi 1 i tes , a posterior utility is 
calculated in the same way as the prior utility (using aq, 
i and 2) substituting posterior probabi 1 it ies Tor prior 
probabilities. This utility provides an ongoing easily 
util izable figure for comparison to alternate model s and 
continuous so rut ini zat ion , 

Discussion 

_ » ~~ ~ _ . — =™= 

While the simplicity of a system utility has merit as 
an overall picture of the functioning of the pool, it is 
likely that data from goals and, subgoals will be most 
effective at directing future efforts and changes in the 
system. A continuous effort to maximize utility through 
maximizing goal attainment will serve both as a diagnostic 
tool and as performance evaluation. 

There are three major problems of the present proposal, 
all of which are not uncommon to evaluation research. 
Because one cannot readily manipulate changes in subject 
pools, numerous potential confounds and sources of variation 
exist. Characteristics of students, attitudes and values of 
experimenters, and quality and diligence of subject usage 
committee evaluations can vary from one year to the nust 
independent of changes in the system* The experimenters and 
subject usage eommitteess could become increasingly avmre of 
evaluation, However, if such evaluation results in 
improvement simply due to being evaluated, that in itself is 



soma measure of success, 

A second problem lias in measurement. The difficulty 
of assessing items such as the advancement of science is 
likely to introduce a large level of error. With more 
evaluation and increased sophistication , such evaluations 
would become increasingly valuable and bather research 
designs possible* 

However , a third problem frequently encountered by 
evaluation researchers is that those whose program is 
evaluated tend to devalue evaluations whose results are 
inconsistent with their personal beliefs* Good evaluation 
research will tell them that their opinion was correct while 
bad or invalid evaluation research will tell them they are 
wrong, 

While early findings in evaluating introductory 
psychology pools can be ekpeoted to have weaknesses both in 
sophistication and level of acceptance, such an endeavor 
would seem to be long overdue, Our subject pools are an 
integral part of our science* as evidenced by their 
representation in the journals* It is time v/e went beyond 
the level of spot criticisms and proceed with detailed 
examination of the process. Only then can v^e confidently 
incorporate these findings in future decisions. 
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